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ABSTRACT 

The regression-discontinuity approach to evaluating 
educational programs is reviewed, and regression-discontinuity 
post-program mean differences under various conditions are discussed. 
The regression-discontinuity design is used to determine whether 
post-program differences exist between an experimental program and a 
control group. The difference between program and comparison vjroup 
regression lines is tested at the cutoff score point. The difference 
between regression lines at the cutoff score point is tested for 
significance against the null hypothesis. Factors that affect 
regression-discontinuity program outcome/effect interpretations 
relate to: (1) measurement error and reliability of assignment 
measures; (2) selection bias and a valid statistical nodel; and (3) 
program outcome and cutoff score placement. Specific issues related 
to the cutoff score — selection, placement, and adherence to the 
cutoff criterion — combined with the reliability of the pretest 
variable and selection bias must be considered when conducting 
regression-discontinuity analysis. Five ta)^les and one figure present 
simulated data from examples. An appendix contains the simulation 
program used, and a 25-item list of references is included. (SLD) 
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Factors Affecting Regression-Discontinuity 



Introduction 



The regression-discontinuity approach is one of the strongest 
methodological alternatives to randomized experiments when 
conducting an evaluation of educational programs.^ Thistlethwaite 
and Campbell (1960) first proposed regression-discontinuity to 
avoid problems inherent in ex post facto experiments requiring 
matching to equate experimental and control groups due to: (a) 
differential regression toward the mean; (b) incomplete matching 
because of failure to identify or include relevant variables; or 
(c) research settings where the investigator was unable to randomly 
assign subjects to experimental and control groups. Bottenberg and 
Ward (1963) presented regression-discontinuity as a type of 
regression analysis involving two mutually exclusive groups. 
Campbell (1969) later clarified the importance of regression- 
discontinuity in evaluating the effectiveness of social programs in 
::onsideration of the political setting of program evaluations 
(evaluation in a social context often forbids random assignment to 
experimental and control groups, involves the allocation of scarce 
resources, or is based on merit or need) . It is not surprising 
therefore to find that during the 1970' s numerous Title I 
compensatory education programs were evaluated usinq the 
regression-discontinuity approach . 

Boruch and Gomez (1977) first proposed a theory of measurement 
in field evaluation where they examined measurement theory and the 
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design of experimental and qpaasi -experimental evaluations with 
issues related to the reliability of the dependent variable and the 
measurement of the treatment variable.^ Trochim (1984) elaborated 
the regression-discontinuity approach as a research design for 
program evaluation with the provision of services based upon a 
cutoff score on an assessment instrument.^ The accuracy in cutoff 
score determination however affects the assignment of individuals 
to groups (Mills et al . , 1991/ Geisinger, 1991).^ 

The regression-discontinuity research design is a member of a 
larger group of quasi-experimental designs commonly referred to as 
pretest-posttest designs. A basic type of regression-discontinuity 
design requires a preprogram measure^ a posttest program measure, 
and a measure that describes the assignment status of the persons 
(received program or did not receive program) . The regression- 
discontinuity design is distinguished from the other pretest- 
posttest designs by its' assignment strategy. Basically, all 
persons are assigned to a program or comparison group on the basis 
of a cutoff score on the preprogram measure. Persons scoring on 
one side are assigned to the program while persons scoring on the 
other side are assigned to the comparison group. The regressions- 
discontinuity design is very useful when researching programs or 
procedures that are given on the basis of need or merit. The 
postprogram measure reflects the effect of the program or 
procedure . 

Many applications of the regression-discontinuity approach and 
issues related to using the technique can be found in the research 



literature. Rubin (1974, 1977) provides a general discussion. 
Berk and Rauma (1983) used the approach in evaluating crime control 
program effectiveness. King and Roblyer (1984) presented 
alternative designs for evaluating computer-based instruction in 
which they recommended using regression-discontinuity when a non- 
treatment control group needs to be formed based upon a pretest. 
Visser and De Leeuw (1S^84) described a maximum likelihood 
generalized regression-discontinuity design wherein differences in 
variance/covariance are considered, multiple pretests and posttests 
can be used (multivariate) , and more than two groups can be 
involved. ^ Stanley and Robinson (198 6) described the use of 
combining multiple criteria using standard score^^ for program 
.<?election in programs for the gifted.^ Roj:;>inscn and Stanley (1989) 
also evaluated a gifted mathematics program where identification 
for selection was based upon multiple criteria. ^ 

The regression-discontinuity approach may le=id to erroneous 
inferences about program effects (Staxiley & Robinson, 1990) . When 
the independent variable of a regression analysis contains 
measurement error, the ordinary least squares estimation procedure 
is biased (Fuller & Hidiroglou, 1978; Fuller, 1987) . Although the 
bias of ordinary least squares regression in the presence of 
fallible variables is well known, the impact of measurement error 
and other factors in regression-discontinuity analysis on the 
interpretation of program effectiveness is not. 

Several states are faced with setting standards and 
establishing cutoff scores for determining school accountability 
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and performance. District-wide educational programs may use the 
same test for pre and post testing^ use different tests^ or a 
composite of several tests as the preprogram measure^ with a 
different measure as the posttest. When testing is used to 
determine who will be admitted to a program or pass a certain grade 
levels the cutoff score becomes important in determining outcomes. 
The regression-discontinuity design is a viable approach for 
assessing program improvement ^ but certain issues regarding the 
effect of unreliable preprogram assessment instruments^ selection 
bias^ and the placement of the cutoff score on program outcome 
interpretation needs to be further investigated. 

The focus of this study therefore is to provide an overview of 
the regression-discontinuity approach and present regression- 
discontinuity post program mean differences under these varying 
conditions, thus explaining how certain factors affect regression- 
discontinuity outcome interpretations. These factors fall under 
the general concerns of: (1) group assigrxment relative to a cutoff 
score; (2) correct model specification; and (3) absence of 
functional discontinuities^ all three being related to the 
distinctive feature of the regression-discontinuity design, nameJy, 
the assignment to a condition (program or comparison group) solely 
on the basis of a cutoff score on a preprogram measure. 
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Methods and Procedures 



The regression-discontinuity design is used to determine 
whether post program differences exist between an experimental 



between the program and comparison group regression lines is tested 
at the cutoff score point . The difference between regression lines 
at the cutoff score point is tested for significance against the 
null hypothesis, : = 0 • 

Regression-discontinuity Simulated Example 

The basic regression-discontinuity design may be expressed as: 



program group and a comparison control group. 



The difference 



Y = 



+ bi Z + b: X + € 



Where : 



X = outcome variable (posttest or policy variable) 



Z = treatment variable (dummy coded; l=program/0=comparison) 



X = identification variable (pretest; assessment instrument) 



r = error vector 



estimated sample regression weights . 



The present study used a simulated data set to present the 
basic regression-discontinuity design (Trochim, 1984) . The SPSS-PC 
program is in the Appendix. The assignment to program and 



comparison groups, treatment variable Z, was based on a cutoff 
score using an identification variable, X (no "fuzzy" criteria was 
employed) . The outcome variable, X, is a score indicating the 
effect of each student's participation or non-participation in a 
program. The simulation program generated 500 pre- and posttest 
scores with a "true" score of 50. Error was then added to pre- and 
posttest scores. In addition, posttest scores received a 10 point 
program effect. The program also subtracted the cutoff score from 
each pretest score creating a new variable, NEWX, which when 
included in the analysis set bj = 0 and resulted in b = Y , , with 

o control " *^ waa 

^1 = Yp^ograa " ^control (program effect) . 

In the simulation program, b„ = 50; bj = 10; and ba = 0 . The 
regression weights will vary, but should remain within a 95 % 
confidence interval of the true value which can be calculated using 
the standard error of the regression weights. Results are 
presented in Tables 1-4 and Figure 1 . The simulated data 
analysis indicated a significant 10 point program effect as 
expected. 



Insert Tables 1-4 Here 



Insert Figure 1 Here 
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In the regression-discontinuity design, b, , is the parameter 
that indicates treatment effect or whether interpretation leads to 
a finding that the program was effective. If = 0, there is no 
treatment effect; when b^ is positive, the program had a beneficial 
effect; and when b^ is negative, the program had a negative effect. 
What factors affect b^ beyond the normal regression-discontinuity 
design assumptions? 

The problem inherent in a regression-discontinuity approach is 
that bias in b2 will effect b^ (Stanley & Robinson, 1990) . However 
ir the basic simulated example, bj = 0, therefore the factors which 
affect bi were not present. The reliability of the pretest 
variable, the correlation between the pretest and the group 
assignment variable {selection bias) ^ and the choice of a cutoff 
score are factors that v*\ffect b^. 

The first concern is that X, the identification variable, is 
not directly known and must be observed with measurement error for 
each student. The victual observed value of the identification 
variable for each student is: 

X, - + u, 

Where : 

Xi - individual observed pretest score 
Ti =^ individual "true" pretest score 
- individual measurement error . 

Q 
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The second concern is that the correlation between the pretest 
variable (X) and the group assignment variable (Z) n.ay be low or 
negative. This correlation is also affected by a third concern, 
the choice of a cutoff score which determines the assignment to the 
treatment variable (Z) and the ratio of the standard deviations of 
X and Z . 

Example with Mathematically Gifted Students 
Subjects 

Subjects used in the analysis were students in the first class 
of the Texas Academy jf Mathematics and Science (TAMS) . TAMS is a 
two-year, early-admissions program for students who are 
particularly talented in science and mathematics (Lupkowski & 
Schumacker, 1991) . Participants enter TAMS after their sophomore 
year in high school and take their 3 ast two years of high school 
and their first two years of college concurrently in residence on 
the campus of the University of North Texas. Participants are not 
given enriched high-schrol courses, rather they take college 
courses taught by regular college faculty. Students who adjust to 
the college curriculum may take additional courses in elective 
areas or advanced mathematics and science classes. 

The first TAMS's class attended the University of North Texas 
in the Fall of 1988. Some students found that early-college 
entrance was not appropriate for them and decided to leave the 
program. Other students were asked to leave by TAMS' 3 staff 

ERIC ^ ^ 



because of behavior or academic problems. There were sixty-six 
students with both pretest and postest data which comprised the 
example data set. 

Preprogram Identification Instrument 

As part of the identification process for admissions to TAMS, 
applicants submit Scholastic Aptitude Test (SAT) scores. The SAT~ 
Mathematic score and SAT-Verbal score were combined and used in the 
analysis (pretest, X variable) . The average combined SAT score for 
all students was 1170 (s = 113; range = 940 to 1450) . The combined 
SAT internal consistency reliability used in the study was .92 
(Kilpatrick, 1980) . 

Post program outcome measure 

The criterion scores (posttest, Y variable) were the students 
overall grade-point average after four semesters in college. The 
average reflects the average of all courses the students took while 
at the university. The grading criteria was : a == 4 pts, B = 3 
pts, C = 2 pts, and D = 1 pt . The mean grade point average was 
3.03 (s == .65; range = 1.81 to 4.00). 

Analysis 

Sample estimates fo^^ b^, b^, h^r r^^, s^, and were calcnlaf^d 
using cutoff scores of 1100, 1150, 1200, and 1250 on the SAT 
pretest measure. Also, corrected estimates of the OLS regression 
weights were calculated as follows: 
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5,=£, [(l-r;,)/(r,,,-ri,)] 
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Where : 

^if^2f^o ^ corrected sample regression weights; 

61,62 = original sample regression weights; 

r^^ = reliability coefficient of X; 

r^^ = correlation of X and Z; 

s^^s^ = standard deviation of X and 

The location of the cutoff score, however, adds another dimension 
to the interpretation of 61 beyond the effects of reliability of 
measurement and selection bias. Which cutoff score maximizes the 
program effect difference between comparison and program 
participants ? 

Table 5 indicates the effect of various cutoff scores, given 
high reliability and minimal selection bias, on the corrected 
sample regression weight, b^ . The program effect or the mean 
difference between the program and comparison group would equal the 
regression weight, b^^, when the correction factor equals zero 
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(perfect reliability and no selection bias) or bj = 0. However, 
when bias in affects bi, the cutoff value influences where CF 
approaches zero and maximum program effects are indicated. A high 
adjusted value would be another indicator of the best cutoff 
score that maximizes program effects. 



Insert Table 5 Here 



Summary 

Factors which afffect regression-discontinuity program effect 
interpretation summarily relate to: 

(a) measurement error - reliability of assignment measures 

(b) selection bias - valid statistical model 

(c) program outcome - cutoff score placement 

The specific issues surrounding the cutoff score: (1) selection of 
the cutoff score; (2) placement of the cutoff score; and (3) 
adherence to the cutoff criterxon, combined with the reliability of 
the pretest variable and the effect of assignment problems 
(selection bias) on estimates of differences in gain, should be 
considered when conducting regression-discontinuity analysis. 
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Notes 

1. Assumptions generally specified in conducting regression- 
discontinuity analysis are: (a) no mis assignment due to cutoff 
selection and placement; (b) statistical model correctly specified 
(linear, quadratic, cubic, etc.); (c) sample size sufficient to 
estimate regression lines; (d) both groups (program and comparison) 
have a common pretest measure or weighted set of pretest measures; 
and (e) all program subjects receive the same amount of treatment, 
for example coursework credit hours. 

2. Educational researchers have also been interested in the effect 
that measurement error has had on various aspects of statistics 
(Sutcliffe, 19b8; Meredith, 1964; Cochran, 1968; Cleary, 1969; 
Subkoviak & Levin, 1977) . 

3. The "fuzzy" regression factor concerns itself with the lack of 
a completely known criteria for assignment of subjects to groups. 
In this study, specific cutoff scores were set and assumed some 
application of a judgemental standard setting method, although 
these methods in and of themselves are controversial (Mills, et 
al., 1991; Geisinger, 1991). 

4. The selection bias factor concerns itself with the lack of 
random assignment of subjects to groups. Selection bias occurs in 
regression-discontinuity when the determinant of program outcome is 
correlated with program participation. This is typically accepted 
as occurring when a misspecif ication of the statistical model 
occurs . 
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5. The author suggests another possible approach using structural 
equation modeling or factor analysis which would use multiple 
pretest variables to create a Pretest Factor and multiple posttest 
variables to create a Posttest Factor. The equation would then 
become : 

Fpo.t = + bi Fp,. + b2 Z + e 

Where : 

^po.t ~ factor score of individual i based on 

multiple Y variables; 
Fp^^ = factor score of individual i based on 

multiple X variables; 
Z = group assignment based upon cutoff 

score on F 
e = error vector; and 

b^r ^ir bj = sample regression estimates. 

6. If two tests are used in conjunction as an assignment variable, 
the reliabilities of the two tests can be pooled using the 
following formula : 
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7. The lowest reliability value possible which would still achieve 
a significant program effect difference at the .05 level of 
significance can be obtained by solving the following equation for 



1 .9esg =B^-[r^^{l-r^^) !i^^-t\^) ]B^{sJsJ 
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Table 1: Simulated data means, standard deviations, 
and correlations (n = 500) . 



Correlations 
X 
Y 
Z 

Mean 

S 



X 

1.00 
.78** 
.79** 
50.00 
1.00 



1.00 
. 98** 
55.13 
5. 19 



1.00 
.50 
.50 



Note: 1-tailed significance: ** = .001 



Table 2: Simulated data pretest means and 
standard deviations by group. 



Group 



Mean 



Std Dev 



Cases 



Comparison 

Control 



49. 17 

50 . 82 



57 

65 



249 
251 
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Table 3: Simulated data posttest means and 
standard deviations by group. 

Group Mean Std Dev Cases 

Comparison 50.01 .94 249 

Control 60.20 .98 251 



Table 4: Simulated data regression-discontinuity 
analysis . 



Variable b SE^, 



Z 10.22 .14 

NEWX -.01 .06 

Intercept 4 9.99 .08 



Note: = .96 
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Table 5: Mathematically Gifted Program Effect given selected pretest cutoff 
values . 



cutoff 




(l-r„/r„-r2„) 




b2 


CF 


bx 






Adj 


1100 


.74 


. 178 


257 


. 003 


. 10 


. 12 


. 02 


.73 


.37 


1150 


.79 


.211 


226 


-.001 


. 03 


1.22 


1 .19 


1.08 


.71 


1200 


.79 


.211 


235 


. 002 


. 08 


.42 


. 34 


. 82 


.41 


1250 


.74 


. 178 


263 


. 004 


. 14 


-. 14 


. 0 


. 61 


.37 



Note: 



C^"-^ t^xz ( 1 -^xx) / ( ^xx--^L) ] 4 ( 3js^) 



5^ ^B^-CF 



62 



Y =60 

program 



58 



56 



(Posttest ) 



54 



52 



^control 50 



48 



113 11 

214213 11 11 
23311131212 1 11 1 
335322313111 22 22 11 
23314123 121 11 111 11 
1134334343212311222 
112 61 3112 1121 21 
212121321 3 4 1 

11 121 1 222 
11 111112 21 
2 1 



11 1 1 
1 111 

I 1 21 121 1 
111 21 122 121 

11123211 214121 2412 
11 2 211151243336131 
111121241322435311 
11 23 21112412153351 
1 2 12121121122 4121 
1 12142 1 1122 

II 111 2 
2 1 2 1 

1 2 



47 



48 



4 9 



50 



51 



52 



53 



(Pretest) 
Cutoff = 50 
Figure 1: Plot of Y with X 
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APPENDIX: SPSS-PC SIMULATION PROGRAM 



SET SCREEN=ON. 

DATA LIST FREEFIELD / ID. 

* Calculate true score and error for pre and post tests. 

COMPUTE TRUE = 50. 
COMPUTE XERROR = NORMAL (1) . 
COMPUTE YERROR = NORMAL (1) . 

* Calculate pretest scores with error. 
COMPUTE X = TRUE + XERROR. 

* Assign subjects to groups based on pretest score. 

IF (X LT 50) Z = 0. 
IF (X GE 50) Z = 1. 

* Calculate post scores with 10 point effect for program subjects, 
COMPUTE Y = TRUE + YERROR + (10 * Z) . 

* Calculate new pretest score with value of zero at cutoff point. 
COMPUTE NEWX = X - 50. 

BEGIN DATA. 

< Enter numbers 1 - 500 in freefield format here > 
END DATA. 

CORRELATION X Y Z / STATISTICS = 1. 
PLOT PLOT = X WITH Y. 
MEANS TABLES = X Y BY Z . 
REGRESSION VARIABLES = Y NEWX Z/ 

DEPENDENT = Y / 

METHOD = ENTER. 
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